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ABSTRACT 

In an effort to meet the accrediting criteria of the 
Southern Association of Schools and Colleges, South Carolina 
Technical College undertook a comprehensive external evaluation of 
its continuing education program. Both process and products 
associated with the continuing education program were assessed using 
newly-generated and existing data. Additionally, a practical model of 
program evaluation was developed to ensure that the institution would 
continue to meet accrediting requirements. The process used in 
developing the model included a survey of faculty and administx'^tors , 
a sampling of course evaluations, analysis of qualitative information 
in the institution's database, and examination of advisory committee 
meeting minutes. The resulting model includes these components: 
establishment of mission, goals, and standards and continual 
documentation of related meetings; determination of at least one 
outcome standard for each goal; assessment of program ef f ectiveness 
in terms of these standards, based on the above process; 
implementation of needed changes; and review of the mission, goals, 
and standards. It is concluded that the model offers a well-rounded 
and thorough approach to assessment satisfactory to any accrediting 
body. (Contains 10 references.) (Author/MSE) 
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Abstract 

Since 1985 assessment and accountability have been part of the accreditation or re- 
accreditation considerations of the various regional accrediting bodies. The Southern Association 
of Colleges and Schools (SACS) was among the first to incorporate assessment into their criteria. 
As a result SACS standards have lead the others in their assessment expectations. In an effort to 
meet SACS criteria a comprehensive evaluation of the continuing education program of a two- 
year Technical College in South Carolina was undertaken by two external evaluators. Both the 
process and the products associated with the continuing education program were assessed by 
using newly generated and existing data. Additionally, a practical model was developed to ensure 
the institution would continue to meet SACS requirements. The methodology and proposed 
model should prove useful to two- and four-year colleges and universities desiring to enhance 
their programs and to meet evaluation requirements of any accrediting body. 
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Assessing Continuing Education: Difficult but not Impossible 

Introduction 

In 1985, Dr. William Bennett, who was Secretary of Education at the time, commented in 
the Chronicle of Higher Education , that “Colleges should state their goals, measure their success 
in meeting those goals and make the results available to everyone.” This comment sounds 
relatively innocuous and would probably elicit agreement from the general public. However, 
Secretary Bennett went on to say that “If institutions don’t assess their own performance, others- 
either states or commercial outfits-will most likely do it for them” (1985, p. 25). Not 
surprisingly, most of the assumptions underlying this emphasis made their way into th e Federal 
Re gister (1988) as Part 602 of the Secretary’s Procedures and Criteria for Recognition of 
Accrediting Agencies. This move firmly implanted assessment and accountability in the 
accreditation or re-accreditation considerations of the various regional accrediting bodies. One of 
the first and most stringent accrediting Bodies to fervently apply the Secretary’s Criteria was the 
Southern Association of Colleges and Schools (SACS). SACS is the accrediting body responsible 
for the colleges and universities in the Southeast and their requirements fonned the basis for the 
evaluation described below. Because of the emphasis placed by SACS on the assessment and 
evaluation process, any institution which meets SACS criteria should have little difficulty in 
meeting the requirements of any of the 7 other regional accrediting bodies. 

The Evaluation 

An evaluation of the Continuing Education Program at a South Carolina Technical 
College was undertaken by two external evaluators. It was the evaluators’ understanding that this 
was to be the first comprehensive evaluation of the program. The evaluation was requested to 
assist the college in preparation for a SACS re-accreditation visit, provide administrators with 
feedback on the effectiveness of the continuing education program, and to provide a model for 
continuous evaluation and operations. The process used to conduct the baseline evaluation and 
develop the evaluation model is offered as a realistic approach to begin or refine program 
assessment methodology. 
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Data Gathering 

In initial discussions with representatives of the Technical College, it was agreed that 
multiple measures would be used to determine the effectiveness of the program and to develop a 
continuous evaluation model. The use of multiple measures in assessment has been advocated by 
numerous authors (Bogue, 1992; Ewell, 1983; Nichols, 1991; Underwood, 1991) and 
Underwood specifically recommends the identification and use of existing assessment data. The 
measures selected for this evaluation included generation of new data through surveys and 
analysis of existing data such as database elements, committee minutes, and historical course 
evaluations. 

All agreed that advisory committee members and department heads would be surveyed 
regarding the effectiveness of Continuing Education in meeting its mission and to identify specific 
strengths and weaknesses. In addition to the surveys, the evaluators agreed to review a sample of 
course evaluations and advisory committee minutes from a one year period, and to explore the 
current database to determine if meaningful data existed there. Once the evaluation was 
underway the Technical College representatives suggested the evaluation should also include 
surveying specific faculty with knowledge of both credit and non-credit courses. As a result, 
three separate surveys were designed and administered. One follow-up mailing was used with 
each survey group. 

The Technical College was made up of six separate divisions, each headed by a Director 
and each receiving input from an Advisory Committee made up of people from business and 
industry, community leaders, etc. The departments included Business and Human Services, 
Enterprise Development, Health Care, Career Development, Licensing and Recertification, and 
Applied Microelectronics. 

The three groups (Department Heads, Directors, and Faculty with knowledge of both 
credit and non-credit programs) were surveyed using consistent items so that comparisons of 
responses could be made. The survey items were developed by the evaluators and were provided 
to the Dean of Continuing Education for concurrence prior to finalizing the instruments. The 
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surveys were formatted to look like other surveys used by the Technical College to aid in the 
consistency of response and to avoid confusion about whether the survey was “official”. The 
respondents were assured of confidentiality and to further enhance the confidentiality aspect, the 
return address envelopes were addressed to the evaluators, not to the Technical College. 

Survey Results 

The overall response rates varied slightly by group and ranged from 65% to 75%. 
Response categories included Advisory Committee Process, Instruction, Instruction and Service, 
and Facilities and Equipment. The specific responses were reported to the Technical College; 
however, for the purpose of this report it will suffice to say the responses were overwhelmingly 
positive with average responses varying by group. Generally, Department Director responses 
were less positive than those of either the Advisory Committee Members or the Faculty. The 
lowest ratings went to Facilities and Equipment with the advisory Committee Process receiving 
the highest ratings. When asked to compare credit and non-credit courses faculty rated the non- 
credit courses slightly higher with a significant difference in Facilities and Equipment 
Course Evaluation Findings 

In an effort to use existing data as part of the Continuing Education evaluation at the 
Technical College, course evaluations were sampled. For the purposes of the evaluation, the 
records for all non-credit courses taught over a one-year period were identified. All courses 
taught between May of 1994 and June 1995 were used to provide the analysis which follows. 
One thousand thirty-one courses were taught during the period used for the evaluation. A 
stratified random sample was generated using a table of random numbers (Kerlinger, 1973) to 
identify the courses which would be checked for evaluations. Each of the six divisions was used 
as a stratum and was separately sampled at a 10% rale. Su^atifying the sample assured that all of 
the divisions were represented at the same rate. Since this was an actual random sample, the 
percentages providing evaluatioii records and the percentages in the rating of the courses, should 
be indicative of the entire population of the year being analyzed (Gall, Borg & Gall, 1996). 
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The first observation to be gained from this portion of the evaluation was the divisions 
were not consistent in their use of course ratings. The percentage of courses which had 
evaluations on file ranged from 100% for Applied Microelectronics to 0% for Enterprise 
Development, with an overall rate of 60%. 

Of the courses in the sample, three different types of evaluation forms were used. In one, 
using a four point scale, participants rated the courses on instructor knowledge, instructor 
presentation skills, the overall value of the training, and training facilities. The choices used on 
the scale for this instrument were: Excellent, Good, Average, and Poor. A maximum of 803 
participants rated the courses according to the scale just described. 

In a second set of course ratings, using a different four point scale, participants rated the 
courses on the instructor knowledge, presentation of course material, instructor organizational 
skills, promptness of class meetings, training facility/classroom, tuition appropriateness, and time 
sufficient for the course. The choices used on the scale for the second type of instrument were: 
Excellent, Good, Average, and Fair. A maximum of 143 participants rated the course?> according 
to the scale just described. 

Only 1 1 courses that had ratings used something other than the scales described above. 
Those eleven courses were all from the Microelectronics Division and a short answer, open-ended 
format was used. It is notable that the later, more open-ended, instrument appeared to produce 
the most instructive re.sponses. The summary forms that were found in the files were converted to 
frequencies to provide an indication of how the courses were rated. The overall responses 
gathered from these traditional instruments could be characterized as positive but provided little 
evidence of areas that could be improved. In contrast, the written comments provided on the 
Microelectronics course evaluations suggested participants were satisfied with the offerings but 
gave specific guidance as to areas needing attention. 

Database Report Results 

It is always prudent when ' onducting an evaluation to consider items and methods which 
may already be in existence rather than having to continually develop new sources of data 
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(Underwood, 1991). This approach allows for the allocation of minimal resources while 
providing for the use of multiple measures. Since no single indicator is considered sufficient to 
provide an overall indication of the strength or weakness of a program, evaluators are challenged 
to provide cost-effective means to develop multiple measures. Often the databases which are 
used in everyday operation, not having been designed to be used in a program effectiveness 
scheme, are of limited or no use in providing data on which a program might be evaluated. 
Therefore, the suggestion of the possibility of using the existing database was greeted with 
skepticism when it was first discussed as a possibility in this evaluation because of its recognized 
limitations. 

The evaluators’ analysis of the database yielded nothing directly related to the quality of 
the offerings of the Continuing Education Program, however it was possible to document several 
aspects of mission fulfillment by the Continuing Education Program and to provide information 
relative tc planning efforts. According to Bogue (1992) evidence of mission fulfillment is 
evidence of the quality of an institution. As an example of how that might be demonstrated by the 
database, reference was made to the “Status Report” which was requested by the evaluators to 
allow the selection of a sample of courses for the purpose of checking the evaluations. The status 
report was a printout which identified the following variables: SES, COURSE NUMBER, 
COURSE TYPE, COURSE TITLE, BUILDING CODE, LOCATION, COUNTY, START 
DATE, END DATE, NUMBER OF STUDENTS, CONTACT HOURS, INCOME, INST 
COST, OTHER COSTS, NET, CO-SPONSOR, WHO FOR, TOTAL CRS HOURS, and REC #. 

One aspect of the mission of the Technical College studied was to provide courses that 
meet the needs of business and industry. That could be documented by generating lists on the 
variable WHO FOR. Those lists could be broken out by year and the frequencies of requests 
could be tracked to indicate growth over years as well as additions of new requesters, etc. 
Additionally, if the same requester continues to show up from year-to-year, that would provide a 
strong indication that the training being provided is meeting the client’s need. 
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Another possibility existed since the COURSE NUMBER contains a three digit code 
which identifies the division providing the training. Similarly to the first example, a sort could be 
done by division that would allow for documentation of the efforts of that division in providing 
training. Again, this type of report could be generated annually or more frequently and the 
numbers at each point would provide trend data indicating growth or decline in service provided 
and to whom. 

The LCXIIATTON variable offers another possibility for documenting service. An analysis 
of that variable would provide an indication of the geographic area being served or identify areas 
which might require outreach since there has been no training provided there. 

These examples are not meant to be all inclusive of the possibilitit;S. However, as 
simplistic as these suggestions may appear, they provide a means to document both mission 
fulfillment and to provide information which can be used for planning or for determining if an 
existing plan is being followed. Alone, this will not provide strong enough evidence of the quality 
of the program, however, it is yet another indicator and can add strength to the rest of the 
evaluation or planning effort 
Committee Minutes Results 

Committee minutes, which represented one year of advisory committee meetings, were 
examined to determine if they could be used as part of an evaluation of the Continuing Education 
Program. It did not appear that the minutes provided a strong enough link between the goals of 
the Committee and the outcomes of the Continuing Education Program. However, having the 
committee minutes on file did provide documentation of the level of activity of the comniittee. 
Additionally, it appeared from both the comments on the surveys used as part of this evaluation, 
as well as in a review of the minutes provided, that these advisory committees make suggestions 
and provide expertise which results in changes to courses and offerings. 
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Model Development 

Evaluation of Current Assessment Program 

In developing a model assessment program for the Technical College, the evaluators 
compared and contrasted the current assessment efforts with SACS criteria. Section HI of the 
Criteria for Accreditation of the Southern Association of Colleges and Schools (1992) states that 
“the level of institutional quality depends not only on the educational processes and resources, but 
also on the institution’s successful use of those processes and resources to achieve established 
goals. ...An institution has an obligation [emphasis in original document] to all constituents to 
evaluate effectiveness and to use the results in a broad-based, continuous planning and evaluation 
process” (p. 15). Although no specific format for the planning and evaluation process is 
prescribed, broad based involvement from faculty, staff, students and administration is mandated. 
An effective assessment process must also include: 

1. the establishment of a clearly defined purpose appropriate to collegiate education; 

2. the formulation of educational goals consistent with the institution’s purpose; 

3. the development of procedures for evaluating the extent to which these educational 
goals are being achieved, and 

4. the use of the results of these evaluations to improve institutional programs, 
services and operations. (p.l6) 

The SACS exhortation to formulate educational goals consistent with the institution’s 
purpose requires much greater clarity in the specificity of purpose statements than historically 
provided in either mission or vision statements. Additionally, the SACS contention that planning 
and evaluation processes must be designed to demonstrate that the institution’s purpose and role 
are being fulfilled, indicates that (1) the ultimate determination of institutional effectiveness is in 
relation to the statement of purpose and that (2) demonstration of institutional effectiveness is 
through the accomplishment of goals and expected educational results that are clearly linked to 
the institution’s stated purpose. In the Evaluation section of the Resource Manual (1992) 
institutions are encouraged to: 
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strive not only to use explicit language but also (1) to specify appropriate time frames 
(e.g., for periodic assessment, for achievement of particular goals over several years, etc.); 
(2) to address both minimum standards and targets for excellence; (3) to indicate at what 
level (e.g., program, department, division, institution, etc.) the assessment will be 
conducted and used; and (4) to designate responsibility for implementing evaluation and 
for ensuring appropriate use of evaluation results (pp. 9- 10). 

A review of a draft planning document recently prepared by the Continuing Education Division 
indicated compliance with many of the requirements specified by the SACS Criteria for 
Accreditation (1992). As is often the case, the evaluators found the Technical College to lack 
documentation of a “continuous planning and evaluation process” and the “use of the results of 
the.se evaluations to improve institutional effectiveness”. Also, the institution’s draft document 
did not provide an indication that the planning nrocess had “broad-based involvement of faculty, 
staff, students and administration” as required by SACS (1992). Furthermore, a review of the 
entire., assessment efforts of the Technical College revealed a somewhat typical pattern. Many 
aspects of an effective assessment effort were in place but they were fragmented and poorly 
documented. Additionally, emphasis was being pLced on process and not outcomes. With 
administrators and faculty overwhelmed with the daily demands of their programs it is not 
surprising that their assessment efforts were not comprehensive. The evaluators attempted to 
provide a broad framework for continuous assessment by suggesting ways to integrate effective 
techniques into the routine processes underway at the institution. The following 
recommendations are provided to illustrate the type of integration needed by many programs. 
Specific Assessment Recommendations 

Statement of Purpose: The Continuing Education Division had a clear statement of purpose 
which appeared to be appropriate for the mission of the institution. It was suggested that an 
expedient approach to documenting the required broad-based involvement would be to use 
minutes of meetings, agenda items, or other routine methods. Integrating this evidence into 
existing documentation could be accomplished on an on-going basis with minimal effort. 
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Goals and Standards: The goals of the Division were clearly spelled out and included 
standards. However, the majority of the standards, as listed by the institution, were “process” 
rather than “outcomes” standards. Process goals and standards ai*e necessary since effective 
processes can lead to effective outcomes. However, documenting that a process is in place does 
not provide proof that the result expected from the process is the result actually being obtained. 
For example, the institutions GOAL 1 was, “to assess and meet the lifelong learning needs of the 
service region”. There are actually two commitments made by this goal. One is to assess the 
lifelong learning needs of the service region. This aspect of the goal is process oriented. The 
institution must know what the needs are before they can be met Meeting the needs is the second 
part of the goal and is much more outcome oriented. 

In reviewing the four standards associated with this first goal, it appeared that three were 
process standards inasmuch as they referred to steps to be taken to validate the assessment of 
needs. The outcome portion of the goal, meeting the needs, was not as clearly identified in a 
standard statement. The standard most appropriate to this aspect of the goal was standard four, 
dealing with certification examinations. As it was worded, that standard was process oriented as 
well since it provided that the certification examinations would be “monitored”. An outcome 
standard, if the training is to help participants pass certification examinations, would be that a 
percentage (to be determined by the faculty and administration) of those taking the certification 
examinations would pass. 

The evaluators emphasized the need to have broad-based participation by faculty and 
administration in their development of the statement of purpose, goals and standards. It was 
suggested that this could be documented again through routine minutes of meetings, agenda 
items, or other existing methods. Furthermore, the evaluators advised the institution that written 
documentation would not be sufficient to meet these criteria. The institution should anticipate 
SAC.S reviewers asking faculty whether they had input into the process. 
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Designation of the Responsibility for Evaluation and Use of Evaluation Results : Each of the 
standards in the draft document had identified a position (Director, Dean, etc.) responsible for 
carrying out the evaluation. However, it was not clear who would be responsible for 
documenting the use of the results. The evaluators suggested specifically identifying responsible 
parties. 

Multiple Measures: As has already been mentioned most oif the authorities on assessment and 
evaluation recommend the use of multiple measures (Bogue, 1992; Ewell, 1983; Nichols, 1991; 
Underwood, 1991). A limitation of the standards as they were stated, partially a result of the 
process nature of most of them, was that they consisted of counts or other types of verification 
that a process happened; not that a desired result for which the process was developed was 
actually obtained. 

Unfortunately, outcomes based standards are often more difficult to document and are 
more likely to require the use of multiple measures. For example, if the standard, as in Goal 1 
Standard 2 is to assure at least 40 business/industry visits per year, then the only measure 
necessary is a count of how many visits were made and a comparison with the stated figure of 40. 
However, if the outcomes oriented portion of the GOAL 1 is addressed, the methods are not quite 
so clear. For example, in GOAL 1 the second portion is to meet the lifelong learning needs of the 
service region. This goal would require documentation that the needs have been met. Finding out 
what the needs are and providing appropriate classes are processes which may lead to the 
successful completion of the goal of meeting the needs, however, those processes do not 
document that the needs were in fact met. 

One method, suggested by the evaluators, to address the outcomes’ portion was to use a 
measure of client satisfaction. In other words, ask them. In this case there would be at least three 
clients who would be the focus; (1) the individuals who took the courses, (2) the business or 
industry who requested the courses, and (3) community leaders who have knowledge of the types 
of training needed and the type being provided. 
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In short the evaluators recommended the institution : 

1. Document all faculty and administration meetings to develop or update the purpose statement, 
goals, and standards. 

2. Recognize that the goals and standards document is considered to be a “living” document and 
that it should be updated regularly. The focus could initially be on three or four major goals 
related to the most pressing issues. After a determination of how well those goals are being 
met, the document should be updated with some goals dropped and others added. Over a 
period of years, all major goals could therefore be evaluated without overtaxing the ability to 
collect and analyze data. 

3. For each goal there should probably be at least one outcome standard. 

4. Course evaluations should be used with every course and filed. Some poition of the course 
evaluations should probably be standardized to provide input which can be compared across 
divisions. The standard portion should be designed to collect data specifically related to the 
division goals. For example, if an expected outcome is that the training provided will meet the 
participant’s needs, a question such as ‘To what degree did the course meet your needs?” 
could be asked with a rating scale attached. Although this is self-report data, it provides 
evidence of goal attainment if the goal of the training is to meet participant needs. Additional 
Survey items could gather information relative to specific departmental goals. 

There should be someone specifically charged with analyzing at least a sample of the 
course evaluations on a regular basis, perhaps annually. That person could make 
recommendations for improvements in each of the areas evaluated (or no changes if none are 
needed) at a deparunent or division meeting. The results could be discussed, reported in the 
minutes and filed in such a way as to provide a paper trail documenting that the evaluation 
results were discussed by faculty and administration and the uses made of the information, 
particularly where changes are made as a result. 
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5. The Advisory Committee members should be provided with an evaluation opportunity on a 
regular basis, perhaps every two years. Since the committees are made up of business and 
industry leaders, as well as community members, they can respond as to the types of training 
needed and the types being provided. This would provide input from an additional set of 
clients. Again, the instrument should ask questions which are directly related to the goals of 
the Continuing Education Program. 

The suggestion was made earlier regarding the use of minutes from the Advisory 
Committee meetings. That suggestion bears reinforcement here. The Advisory Committee 
provides input for planning as well as recommendations for program improvement; however, 
the link between those suggestions and the faculty/administration discussion and the 
implementation of the input is not documented clearly enough to show SACS that the link 
exists. 

6. Business and Industry clients who request training should be surveyed on a regular basis. This 
would provide input from the third set of clients. Again, the survey should ask specific 
questions relative to the goals of the Continuing Education Division and possibly some 
questions which are specific lo the department providing the training. A sample could be used 
to cut costs and the survey could be conducted on a three-year basis, to avoid conflicts with 
the schedule for surveying Advisory Committee members. 

Suggested Model Outline 

In an effort to clearly communicate the assessment model advocated for the Technical College, 

the an outline of the model was developed. It is offered here as a generic model that would be 

effective for most institutional programs. 

I. Establish Mission, Goals and Standards 

Continually document meetings related to the mission, goals and standards. 

II. Determine at Least One Outcome Standard for Each Goal 

III. Assess Program Effectiveness Relative to Established Mission, Goals and Standards 
Examples 
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1 . Course Evaluations (Annual Cycle) 

Do course evaluations regularly, file them consistently, ask questions directly related 
to the goals, standardize at least a portion of the evaluation for comparison purposes, 
allow for open-ended comments, review at least a sample annually and bring the 
findings back for discussion at the time of review of the mission, goals and standards 
document. 

2. Advisory Committee Evaluation (2 Year Cycle) 

Survey the Advisory Committee members on a regular basis, perhaps every two years. 
Ask questions specifically related to the mission and goals. Bring the findings back to 
the next review of the mission, goals and standards document. 

3. Constituent Evaluation (3 Year Cycle) 

Survey a sample of constituents, on a regular schedule. Every three years would 
probably be sufficient Bring the findings to the review of the mission, goals, and 
standards document 

4. Administrative Database or Records Evaluation (Annual Cycle) 

Use the database to generate reports which will provide trend data regarding such 
variables as numbers of classes taught, location, for whom, etc. These reports can 
provide additional information for use in determining whether some of the goals are 
being met and also provide the population from which to choose a sample for the 
survey of constituents. 

Implement Needed Changes 

Develop a method to create program improvements resulting from the assessment. 

Review Mission, Goals and Standards 

Set a regular schedule for reviewing the mission, goals and standards document in light of 
information coming in fiom the on-going evaluation; probably annually. 
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Conclusion 

The model described above provides a well-rounded, thorough approach to assessnitnt and 
evaluation that should satisfy SACS or any other accrediting body. Samples could be used to cut 
down on costs and resources, and the schedules of the various surveys can be manipulated to 
avoid becoming overwhelming. It is important to remember that for each goal and standard there 
is an expectation of collection of information and use of results. 
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